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Item-Item Curves and C(inslatent Mental Test Parameter Estimates"^ 

♦ ■ ■ 

. Abstract 



It is shown that empirical mental test P - P plots are appi^oximatej-y 
equal to ' theoretical item-item curves, \at l^east for long testa ad3u.nistere< 



to many people. TKis result is import^t because i"^ leadg to (l) a 
distribution free method fpr estimating points on item-item curves,' 
(2) a general method for defining estimates of item parameters and' (3) the 
only parameter estimation method known to be consistent for some of the 

• V ' ■ . \ 

most important aptitude test models. ^ 



!• Back^found and Motivation 

\The "answer sheets" for tntatiple choice aptitude tests are commonly' 
considered to be generated by a two stage process. First, an examinee 
with some (unobserved) ability 9 is sampled. Second, the various items 
are answered independently so that the conditional probability of a 
randomly selected person with ability 6 having a specified pattern of 
right and wrong answei^s on the first n items of the test can be expressed 
in the form 



n u. , l-u. 

n P.(©) "-[i - P (©)] ^ 
' i=l V 



where u^ = 1 or 0 according to whether item i is correctly answered 
or not and P^(0) is the conditional probability of passing the i -th 
item. 

The conditional passing probabilities' or "item characteristic 

functions" P^^^ are generally assumed to have one of three forms 
t 

V 

(1^ , (Rasch model) P^(©) = P(© - b^)". 

(2) (Two parameter model) P^(9) = P(a^0 - b^) 

(3) , (Guessing model) P^(e) = + (l - c^)P(a^O - b^) 

P is> usually specified to be the logistic functipj/ 

(1 + e ) 
or the noTToal ogive 



-2 

4 



r 
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Our results will be applicable to a much wider class of functions P 
and . 

For a^' discussion of these models and basic parameter estimation 
results^ see Birnbaum (l968)» 

Throughout this paper Ve assume that the distribution of ability is 
unknown. For comments on tjiis point see section II.U.l. 

Our basic result provides a mathematical justification for a 
distribution-free method for estimating points on item-item curves, a 
method which is currently being used [ Levi ne and Saxe, 1976]. (item-item 
curves- are defined below. They play an essential role in the application 
of functional equations and group theoretical methods to psychometric 
problems [Levine, 19TC)>^J.972, i975; Levine and Saxe, 1976].) 

Our results also have implications for parameter estimation, \ 
especially with equation (2) and (3). The extra generality of equations 
(2) and (3) seems to be needed for such important applications as the 
design of optimal tests, computer supported individualized testing and 
the detection' of bias in tests. However, t^e Estimation of item parameters 
( a^,b^, c^ ) is considerably more complicated for (2) and (3) than for (l). 
In fact, it is not kndwn whether any of the parameter estimation procedures 
that are now used (that is, used in applications in which the ability 
distribution is not specified) are consistent. 
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Our results :^an be used to show that a simple, common-sense way to. 
estimate parameters (for (2) as well as (l)) is consistent. The results 
may be applicable to (5) as well, but an elementary step in the 
verification for (5) has not been proved- 

It is unlikely that the estimation technique suggested in this 
paper will be applied in its present form as it uses d^ta inefficiently* 
However, the technique seems intrinsically interesting as an application 
of P - P plots [Gnanadesikan an4 Wilk, I968]. Furthe'rmor.e, it seems 
important to chow that at least one consistent estimation procedure is 
available for the general models. 




II. "dlie Model and an Informal Discussion of the Basic Result 

Since the result is coraplicated, we flf st present the ^ 
centrlal finding informally, in outline form, together with some 
explanatory*' comments . 

» 

II. 1 Model and Notation 



We assume that the "abilities" or 0 'a are^obtained by sampling 
from a population with a continuous density d^^^oted by f . The i -th 



item score random variable u. has the interpretation: u « 1 if 

item i is answered correctly and zero otherwise. is called the 

i -th item characteristic funct'^n. 

We assume that for each n , each vector of zeros and ones 

< V. ,v^, ...,v > and each interval of numbers T . 
1 n 



Problu^ = v^, = v^, ... ^n ' © c T) 



/n ' V 1-v 
n p.(t) ^[1 - p.(t)J ^ f(t) (ft 

^ i=l 



This model will be used throughout the paper. 

II .2 Basic Result 

We assume that each P^ is strictly increasing and^continuous, tljat 
f is continuous and that these functions satisfy technical conditions 
detailed in section III. . 



i. 



Consider any two items, say item 1 and 2. The item-item curve 
for these items is the point set ^ C = C(l,2) given by 

C = ( < x,y > e IfT I for som« t , x = Pj^(t) and y = F^(t)] 

Let d be the usual distance froni a point < x, y > in the plane 
to the curve C , i.e., ; 

if 

M < x,y > ) = inf |/[x - P^(t)f+ [y - V^(t)f 



Denote by z the averace 
n 



n n^^^ i 



This is simply the usual average score or proportion correct score on the 
first n items. 

Assume C is a proportion in the range of each P, . Then it is 

^ i ; ' 

possible to select a sequence of positive numbers s decreasimr to zero 

n ^ 

such that the sequence of points x .y > in the unit square with oo- 

n n 

ordinates given by the conditional expectations * , 

X = e(u_ L |z - ^1 < s ) 

{? ' 1| ; n. ^' n^ 
is eventually close to C in the sense that " 



d( < X ,y > ) > 0 



HQ Significance of Basic Result 

The basic result Ituplles that If a large sample of people Is 
administered an n -Item test, the point In the plane defined by the sample 

conditional proportions * ' ^ 

« 

A • 

= proportion passing item 1 among those with proportion 

correct within s of t 
n ^ 

A 

' ^n proportion passing item 2 among those with proportion 

correct within s of f 
n ^ 

will be very likely to be close to a point on the curve C. For <x .y > 

n , n 

can be made arbitrarily close tfc <x ,y > by taking a Sufficiently lar«e 

n n . 

sample. And '^\>y^^ can be made arbitraily close to a point on the item- 

item curve C by taking Efficiently large n . 

These curves can be used to obtain consistency results. For 

suppose (2) is'^pgrrect and P is, say, the logistic function 
-X ^-1 

y - (l + e ) with continuous inverse 



P"^(y) = -log[ ^ - iJ- 



Then the transformation 



carries C to the straight line 

V 



a^x - = - bs 



and the identif ialale parameters (e.g., the slope ^^/^^ intercept 
(l>2 ■ \)/^2 ^ con^3uted by Cramer's ^ule ^rom any tvoMistinct ' 

v.. ■ 



points on C . Since the rormulan of Cramer's rule are continuousj^ 
a Sequence of poirttn C(^nverging to two distinct points on C coiiid 
be iified to define a nequence of estimates converging to parameter 
values. \y 

The carje on hand is somewhat more complicated. The technical 
assumption made in section III will Imply that if two different pro- 
portions ^ and ^* are used t(^ define two different sequences 

^ ^ ^, ^ 

^ X ,y and x ,y ^ , then both will eventually be arbitrarily 
n n n n 

close to C . Generally neither sequences will converge (see section 
11.^. 5)- But it can be shown that corresponding points in the sequence 
will be "separated," i.e., for some positive € , for all n 

(x - X* ; + (y - y' J . > £ 
. n n ^ n n 

This means that the line connecting (transformed) corresponding points 
will converge to the transformed item-item curve line, and Cramer* s 
rule could be used to define a convergent sequence of parameter 
estimates . ^ 

(We consider only two C *s here to keep the reasoning as simple 
as possible. In practice it would be preferable to use many ^ *s and 
estimat^ parameters by fitting a Islne to transformed < x^,y^ > *s«^ 

The key points of the above scheme for estimating parameters for 
equation (2) are: (i) There is at most one item-item curve passing 
through two distinct points in plane and (ii) a continuous formula 
(Cramer's rule) is available for e3g)ressing Item parameters as functions 

/ 
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of pointfl In tlu"* plan**. 'Hw nchrmr, t^hii.-:, clonrLy could hr uaed for the 
mi>del In which P [t\ Ihr normft 1. M^^ivr, or an^ othrr MUch fiuictlon 
aj^pearln^ In (:'). ^ 

One woiild Like to apply t\\lr* e;:tlmftti(>n flcheme to the commonly used 
model in which P ir> logistic ni\d onc.h P^ Matir.rien (^)- Hut unfor- 
tunately^ I havr not yet, r.ucceeded in verify in^j; that for nomc n there is 
at moHt one Item-item ciirve consiptent with (5) and the Logistic aHSumption 
passing through n diiitinct paint.n in the plane, 'Phis elementary step and 
the exhibiting of a continuous inver.sion formula presently blocks the 
appLication of the basic result to the morit commonly used models the 
Logistic guessing model. 

TLie item-item corves < Pj^P^ ' ^^e interesting in their own 
right since they play a central role in the use of group theoretical 
and F'-^urier methods to analyze test data [Levine and Saxe^ 1976]. The 
hH.;( -^^^-^uLt supports a distribution free method for estimating points 
' n tai .^.urves which is currently being used. (The method is to 
<^^>rp ]te the X ^y 's for many ^ *s and then corr^ute t^e best 



n' n 



f i M : og men otonic function to the ^ * x ,y > 's.) 



n n 



II. k Comments 

1- If f is assumed to be normal, then estimation problems 
can be greatly simplifi^ed. However large scale studies 
have clearly disconfirmed normal and other guesses al|out 
^ the form of f . Consequently we choose to make onj 
plausible regularity assumptions about the form of f . 
2. Our method circumvents the complications of simultaneously 
introducing new ^unknown "item parameters" by increasing test 
r length and unknown abilities d by increasing saraplje size. 
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% Since proport ion correct In a ^fraction, z^^ takes only 

rinltoly many Jifferent valnr::. Coruiftjiient, ly we condition 
' upon being in an iriterv^iL c^>ntaining ^ rather than 

n ^ 

Convergence of proportion correct z^ to ^ does not 
imply convergence of efitimated ability. In fact since 
blocks of easy and hard items may be interspersed even the 
sequence^ / ^ 



e(e|zp^ - 1/2) 

in which proportion correct is constant caru oscillate 
indefinitely. A fortlriori. convergence of ^ to ^ 
^ caanot imply convergence . of either 



or 



Thus nelther.^^ 



nor 



need converge. All that can be proved without further 
assumptions ie that < x^,y^ > is eventually close to 
the curve < P.^Po ^ • 

12 
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Our^l^^^altg are fully distribution free. Tbe basic result 
remain^ valid evendf none .a|||(lie formulas (X), (2) or ^ . 
(3) are valid for any P . / 



III. Technical Details 




In ac^sprd with current practice we will generally use 



n 



rather, than 



(5) 



■€(u I ^ Z u - d 1 s ) 
1 n i — n 



That is, we condition on the average score on the test without the item 

J 

being studied rather than jon the first n items. In applications (k) 
is preferred to (5) because for short tests the quantities 



e(u.'| I i z u, - d < s y 

^ J I n ■ i ^ - n' 



obtained for different items j are markedly interdependent [Lord and 
Novick, 1968^ Theorem 16.4.1] Asymptotically (h) and (5) are the same, 
but some of the. proofs using (k) ar^ simpler". ^ 



III/2 



It is well'^known that "true score" or proportion correct on a very 
long test is very nearly equal to a function of ability* We will use a 

more specific result,, namely that under general conditions, if 

3 



s 



0 and ns' 



n n 
is a constant k such, that 



00 ^ then for a given proportion ^ there 



or ; 

ERIC 
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(6) Prob{K6 ability - 0 < ks I I proportion correct - t| < s ) 



1 



vhere G is the unique solution to the equation 
n 



^ = i Z P. (9 ) . 

1=1 " 

• : . . ■ f ' ' . 

yfThe uniqueness of 0^ will follow from the assumptions to be made 
about the P. * s J 

This result will imply the key part of the basic result. Fpr 
denoting , 



= average score on the first n items of the test. 



except for the first Item 



n 



n 



-Z u 
X 2 1 



and 



,w have 



X s= Rrob(u, = 1 & 
n 1 



n 



n 



+ Probfu, = 1 & I 9 - e*j > ks I |z* - d < s ) 
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where 9 = the ability random variable and , 

■ ' •. n 



0* satisfies Z P. (©*) = t, 
n i=2 ^ " 



Formula (6) implies that the second term tends to zero. The first term 

, ' c ^ ^ 

. Prob{u^ = 1 & I 9 - < ks I iz* - d < ) 
1 ' n' — n| ' n ^' — n 

equals 



n n 



/n n 

P^(0)Prob{|z* - d 1 9 «= ©) f(Q)de 

Prob(|z* - d < s } 

' n ^' — n 



0*-ks 
n n 



But since P^^ vill-be assumed to be continuous, this will be very 
nearly 

P,(0*)Prob{| 9 - 0*|.< ks I |z* - d <^ s ) 
1^ n' n' — n| ' n — n 

vhich by;4(6) is asymptotically equal to , * 

Our assumptions about the P. 's will imply that - 0 — >0 

i 1 ^ n n ^ 



Thus 



- P-lC^^) will tend to zero. Similarly, if 



^n- ^(^I 'l E^K^^^ ^n^ - ^1 ^'J 



then y - P^C©^) also tends to zero, and the distance between < x ,y > 
n d n • n^'^n 



er|c IC 



and curve < P^(©)^P2(®) ^ tend to zero^ Thus to^ establish the . , 

result considered in ^11.2 it is necessary only to more precisely formulate 



and prove (6). 



III>3 



In this section inequality (6) is stated more completely and proved. 



Notation and Assumptions 



f = 



tl 



ability" density, assumed to be continuous • 



[a,b] a fixed interval with a < b . 

" ,1 ♦ ■* - 

k'= ming^|-^ b]"^^^^ ' assumed to be pdteitive. 

= "item characteristic function," assumed to be continuous, 

V 

strictly increasing, differentiabl^- 

\ ' ■ ■ . ■ . 

a = inf inf r , iP'(9) , assumed to be positive. 
I, a , D J X 

3 = sup sup^ c v^i^'C©) > assumed to be finite. 



1 




^8 = "ability" random variable. Has density f . 



U. , i > 1 r= "item score" random^ variable, satisfies 



Prob{u 



i = i| 9 = e) = p^(0) 
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z « "proportion correct'* random variable^ satisfies 
n 

z = — E u, • 

n, n £ i 

e ■= a positive constant* We assume for all n 



^ n 



Asymptotic relation between proportion correct and ability : 

3 y 

If s m 0 and ns j then 

— n n ^ : 

\, 

Probd© - ej > 2sya| Iz^ - i[ < s^) ^0 

Proof: Let ^L (O) denote the conditional expectation 
n 

i 

n 



e(z I e « e) = i s (9) 

^ n' ' n . - i^ 



i=l 



Clearly \x, (o) is continuous, differentiable, strictly increasing arid 



n 



satisfies a < m.'(0) < p for all' n and 0 € [a,bJ 
n 

The conditional variance of z satisfies 

I n 



Var(z I 8 = e) P (©)[i - pje)j Si^ 

n 1 



Using this in Chebyshev's Inequality gives 



(7) PTobdz^ - fei)! > 6} 5 r^-^ 

n n tn t^d. 



I 

i ' ' ' 
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Prom the triangle dnec^i<iality we have 

Consequently .; 



(8) Prob(|z -Cl<s 9=0)< Probdti (©) - z | > | (©) - ^ " sl 9 = «) . 

n . — n n n | . 

For 9 e (a^b) and 0 < s < a|0^ - e| it follows frqm (8), the 
mean value theorem and Chebyshev's inequality that 

(9) Prober - d 1 s| e = e) < [a|0^ - e| - sj"^ . 

For s > 0 , put = (a,b) n [6^ 2s/a,9^ + 2s/af . Then. 
© e^U implies 0 < s < 2s < a|0 - 0 | ,\»-(9). can be used to derive 

'J 

.(10) Prob(|z^ - ^1 < s & e e U^) . ^ 



= J Prob(|z^ -^1 < s| = 0)f(O)d0 



U 

n 



u 

n 



4ns 
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, ''' ' , ■ 

For ye € (a ,b )° and 0 < s < ag/2 

(13,) ■ . Probdz^ - Cl < s|e = ©} 

< Prob{|z^ > - si e «= e) , 

from (8) «^ ' 

< Prob{|z^ - > min | n^(x) - 5| - s| 9 « ©} , . 

x=a,b . 

' since ^ is monotonic ' 
from Chebyshev*s inequalfty and the !» mean value theorem, 

< 1 _i 

since ae > 2s . ^ 

Combining these results gives 

for B < 6ce/2, 2s/a 

(12) Prob{|z^ - Cl' 1 - .9 I > t) 

f ^ . Prob{|z^ - ^1 < s| 9- = ©)f(0)de 

u„n(|e„-e|>t) 



0 
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Prob{ I - Cl < s| 9 - e)f (e)d© ' 



[u^u(a,bf]nCl9^-e|>t) '^^^ 

Finally, we obtifein a bound on Probdz^ - Cl ^ s) as follows: For 
s < 23€ , |9 - 9j< implies 6 e (a,b) and 

Probdz^ - ?l < s) ^ 

> Prob{|z^ -d<s&|9^-9|< s/2p) 

> / s Prob{|z^ - |i^(9)| + I |i^(9) - ^1 < 8|9 ■= e)f(9)dG 

© +s/2p , 
n- / ^ / 

> J . Probdz^ - ^1^(9)1 < s - p|9^ - e||9= ©)f(e)de 

9^+s/2p 

> J Prob{|z^ - Uj9)\'< s/2| 9 = 9)f(9)d9 



e +s/26 
J n- ' ^ 



f(9)d9 
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• ^ - -19-. 
" ^ (1 - ) niin f (©) • s/3 - 

: \ 

Combining this^ result vith (12) gives for s < CU^,/2, Sp/g 

r ■ 

Prob{| 9 - e I > 2s/a| (z \ -' 51 < s) 



^5 

which tends to zero as s r-:^0 and ns >oo , as was to be 

proved. 



2'^ 
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am indebted to Joseph B- Kruskal, Cheryl Reed^ Donald Rubin 
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